1. Technical Field
This invention is directed toward a color calibration technique and a stitching technique for use with an omni-directional camera. More specifically, this invention is directed towards a system and method for improving the color uniformity of a composite image generated from a set of images depicting different but overlapping portions of the same scene. In addition, it includes a context-specific image stitching system and method that uses domain knowledge to improve the quality of image stitching.
2. Background Art
An image mosaic or panorama can be generated by aligning and “stitching” input images that are acquired from a single camera. To create an optimal panoramic stitched image, several types of color calibrations need to be taken into account. Differences in exposure, brightness, contrast and white balance can cause significant discontinuities between images. When acquiring the input images for a panorama from a single camera, the camera is adjusted to have as near as possible the same settings, including color settings, for all images acquired. Nevertheless, there are still sometimes differences in color between images because lighting and other conditions may change over the course of time and adjustments to settings may be necessary when photographing images from different angles or perspectives. Furthermore, even when all other conditions remain constant, settings can be inadvertently changed between pictures. As a result, when the input images are stitched together, the images can exhibit differences in color levels, making the edges or boundaries of the stitching in the composite image obvious to the viewer.
The problem of color differences between input images in a composite, mosaiced image is compounded when multiple cameras are used to acquire the input images. For example, this is the case when multiple cameras are used to generate a panoramic video. Essentially, the creation of a panoramic video entails acquiring multiple videos depicting a full 360-degree view of the surrounding scene. A camera rig, consisting of multiple cameras disposed in a back-to-back fashion, is sometimes employed for the purpose of capturing these videos. There is great difficulty in adjusting the settings of the multiple cameras used in this configuration to have exactly the same color settings. Although the parameters may be identically set, variations in Charge Coupled Devices (CCD), Analog to Digital (A/D) converters, and other analog components still cause significant image variations between cameras. As a result, the stitched or mosaiced composite image will often exhibit distinct “edges” where the different input images overlap due to the different colors of these images.
In addition, while the computer vision community had long ago recognized the importance of geometric calibration in accounting for geometric distortions, work on multi-camera radiometric calibration has been limited. Recently, much work has been done in the direction of geometric registration [3], in making panoramas generated by stitching individual images look seamless. In some cases the color uniformity is taken for granted, something that is not always true as discussed above. Others have used techniques to make the change gradual rather than abrupt, techniques known as feathering or blending [3]. Additionally, some research has been done to analyze, classify and correct photometric non-uniformities in and across tilted projectors [2]. The work that has been done in the field of radiometric calibration has mainly dealt with noise estimation and modeling [5, 4] or with single camera self calibration [1]. Radiometric distortion, the effects of lens vignetting, needs to be accounted for to ensure uniform intensity across panoramic images.
In addition to the problem of color differences, other factors affect the desired seamlessness of a mosaiced image. For example, when cameras are arranged in the back-to-back circular fashion discussed above and shown in FIG. 2, the center of projection in each camera is the point between the CCD and outer lens such that rotations about this point result in a pure rotation transformation on images captured (i.e., no motion parallax is induced). The distance between these centers of projection is labeled D. If D were zero, then stitching the images together is relatively easy. However, in some omni-directional camera configurations, D is much greater than zero, which results in motion parallax in the captured images. As a result, stitching the images together seamlessly requires knowing the depth from the cameras to the object being stitched. Computing reliable object depth with stereo image pairs is a very difficult problem, and has not been solved generally (for arbitrary objects, texture, lighting, etc.).
It is noted that in the preceding paragraphs, as well as in the remainder of this specification, the description refers to various individual publications identified by a numeric designator contained within a pair of brackets. For example, such a reference may be identified by reciting, “reference [1]” or simply “[1]”. A listing of the publications corresponding to each designator can be found at the end of the Detailed Description section.